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According  to  the  U.S.  General  Accounting  Office,  the  U.S.  Department  of 
Defense  (DoD)  annually  spends  over  $24  billion  to  maintain  and  develop 
software  for  its  weapons,  command  and  control,  and  other  automated 
information  systems.  DoD  sees  software  reuse  as  a  means  for  better 
managing  these  costs  and  increasing  the  quality  of  the  software  products 
it  acquires  and  maintains.  However,  DoD  efforts  to  encourage  its 
suppliers  to  adopt  software  reuse  have  met  with  a  number  of  acquisition- 
related  barriers.  Issues  that  need  to  be  resolved  to  alleviate  these  barriers 
include: 

•  ensuring  that  suppliers  will  apply  reuse  to  reduce  cost  and 
increase  quality  and  timeliness 

•  determining  the  extent  of  supplier  capability  to  effectively  apply 
reuse  technology 

•  avoiding  the  use  of  specifications  that  impose  non-essential  or 
counterproductive  requirements  on  customers  and  suppliers 

•  achieving  "grass  roots"  support  from  the  acquisition  and  vendor 
communities 

Introduction 

To  gain  insights,  DoD’s  Advanced  Research  Projects  Agency  (ARP A) 
Software  Technology  for  Adaptable  Reliable  Systems  (STARS)  program 
funded  a  case  study  of  Bell  Canada’s  (Bell)  efforts  to  improve  software 
quality  through  a  joint  effort  with  its  principal  suppliers.  Bell  is  the  largest 
Canadian  supplier  of  telecommunication  services,  securing  about  sixty 
percent  of  the  Canadian  market.  Bell  provides  voice,  data,  and  image 
services,  and  the  management  and  operating  systems  required  to  sustain 
them.  These  services  are  based  on  technologies  primarily  provided  by 
Northern  Telecom  (NT)  and  its  research  and  development  arm.  Bell 
Northern  Research  (BNR). 

Bell  is  a  commercial  organization  that  acquires  software  for  systems  that 
resemble  those  of  DoD.  For  example.  Bell's 

•  end-to-end  service  depends  on  software  systems  that  are  large, 
critical,  complex  and  subject  to  rapid  changes 

•  major  switching  systems  contain  more  than  10  million  lines  of 
code,  each  with  a  different  combination  of  features 

•  requirements  expand  continuously  —  twice-yearly  updates  can 
contain  up  to  400  new  features,  each  with  about  5,000  lines  of 
code 

•  updates  must  not  impact  availability  —  systems  must  be 
available  all  but  two  hours  in  40  years  of  operation 


WHAT  IS  TRILUUM? 

Bell's  software  capability  improvement  model  and  methodology 
(TRILIJUM)  includes  software  reuse  as  a  significant  evaluative  factor. 
TRILUUM  incorporates  software  reuse  into  a  framework  that  stresses 
improving  supplier  responsiveness,  increasing  quality  and  reducing  cost. 
TRILUUM ,  which  was  developed  jointly  with  Bell's  principal  suppliers, 
NT  and  BNR,  adapts  DoD's  Capability  Maturity  Model  (CMM)  to  a 
commercial  setting  by  adding  a  strong  customer  focus,  a  product 
perspective  and  technological  maturity. 

TRILUUM  incorporates  quality  management  practices,  industry 
standards  (such  as  ISO  9000),  and  requires  the  involvement  of  the 
customer,  who  adds  specific  requirements  to  each  assessment.  Like  the 
CMM,  the  TRILLIUM  model  consists  of  a  series  of  levels  which  represent 
an  organization's  software  development  maturity.  At  each  level  there  are 
key  work  practices  which  must  be  enacted  to  advance  to  the  next 
capability  maturity  level. 

Objectives,  Scope  and  Methodology 

The  objectives  of  this  case  study  are  to;  (i)  determine  the  process  and 
rationale  that  led  Bell  Canada,  Northern  Telecom,  and  Bell  Northern 
Research  to  require  and  include  reuse  in  their  criteria  for  evaluating 
software  capability;  (ii)  gauge  the  level  of  internal  and  external  support  for 
including  reuse  as  a  critical  criterion  in  Bell  Canada’s  software  supplier 
quality  evaluations;  (iii)  discern  the  impact  of  TRILUUM  and  its 
associated  reuse  practices;  and  (iv)  describe  similarities  and  differences 
between  Bell  Canada  and  DoD  in  purchasing  and  developing  software 
products. 

This  case  study  was  prepared  by  Applied  Expertise,  Inc.  for  DoD's 
ARPA/STARS  program  under  a  sub-contract  to  IBM  Federal  Systems 
Company.  The  study  is  the  product  of  three  months  of  research,  including 
on-site  visits  and  face-to-face  interviews  with  the  developers  and 
implementors  of  Bell's  software  capability  improvement  model.  We 
caution  that  information  used  in  the  preparation  of  the  case  study  was 
acquired  primarily  from  individuals  directly  responsible  for  the 
development  and  implementation  of  TRILLIUM. 


Study  Results 

r/f/LL/t/M  SOFTWARE  REUSE  PRACTICES  ARE  BASED  ON  LONG-TERM,  IN 
HOUSE  EXPERIENCE. 

Reuse  has  long  been  a  part  of  software  development  within  the  tri¬ 
corporate  organizations  (Bell,  NT  and  BNR).  Over  fifteen  years  ago.  Bell 
suppliers,  NT  and  BNR,  created  a  product-line  architecture,  development 
process  and  development  environment  to  field  quality  products  within 
reasonable  cost  constraints  for  digital  switching  systems  and  to  meet 
customer  demands  for  continuous  system  upgrades.  According  to  Neil 
Gammage,  Director  of  Synergy  Architecture  for  BNR: 

"The  only  way  to  make  order  of  magnitude  improvements  in  software 
quality  and  developer  productivity  is  to  write  less  code  —  to  reuse. " 

According  to  corporate  managers,  digital  switching  system  size, 
complexity,  domain  stability,  and  long  product  life-cycles  enabled  them  to 
recover  the  added  costs  associated  with  these  reuse  practices. 

The  effectiveness  of  this  approach  is  demonstrated  by  the  fact  that  only 
minor  changes  to  the  architecture  were  required  for  over  fifteen  years. 
During  this  time,  NT  regularly  added  hundreds  of  new  features, 
expanded  into  international  markets,  and  experienced  significant 
increases  in  revenues.  For  example,  NT's  sales  grew  from  less  than  $1 
billion  in  1985  to  over  $8.4  billion  in  1992,  largely  due  to  their  ability  to 
adapt  digital  signal  processing  technologies  to  new  markets. 

Recently,  however,  a  rapidly  expanding  customer  base,  a  continuous 
demand  for  new  features  and  the  requirements  of  new  technologies  have 
outstripped  the  response  capability  of  the  switching  system  architecture, 
process  and  tools.  In  July  1993,  NT  announced  a  second  quarter  1993  loss 
of  $US  1.03  billion.  The  loss  included  charges  for  an  aggressive  software 
re-engineering  and  technology  initiatives,  proposed  to  senior  management 
several  years  ago.  Improving  software  reuse  is  a  key  element  in  this  new 
initiative. 

A  COMMITMENT  TO  CONTINUOUS  QUALITY  IMPROVEMENT  IS  A  KEY  SUCCESS 
FACTOR  FOR  BOTH  TRILLIUM  AND  REUSE. 

Interviewees  stressed  that  effective  software  reuse  cannot  be  achieved  in 
isolation.  TRILLIUM  places  reuse  as  one  of  a  number  of  development 
practices  that  are  necessary  to  ensure  an  organization's  improved  software 
capability.  TRILLIUM  defines  capability  as  "the  ability  of  a  development 
organization  to  deliver  a  software  product  that  meets  customer 
expectations  with  minimal  defects  for  the  lowest  life-cycle  cost,  in  the 
shortest  calendar  time."  Similarly,  close  partnership  between  customer 
and  supplier  is  cited  as  an  important  success  factor. 

REUSE  WITHOUT  APPROPRIATE  MANAGEMENT  IS  INEFHCIENT  AND  RISKY. 

Uncontrolled  "cloning"  can  be  hazardous.  In  1990,  Bell  discovered 
evidence  that  uncontrolled  cloning  [reuse],  was  far  more  prevalent  than 
expected.  While  analyzing  a  large  segment  of  Bell  code.  Bell  managers 
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were  irprised  to  find  numerous  modules  with  the  same  name  and 
slig'i  :ly  different  code,  and  others  with  identical  code  and  slightly 
r’lJerent  names.  Though  the  program  still  worked  properly,  the  design 
inefficiency  and  potential  for  serious  problems  with  bugs  were  clear. 

Use  of  libraries  without  process  and  infrastructure  risks  systematic 
software  flaws.  Bell's  Jean  Normand  Drouin,  who  has  conducted  over 
fifteen  TRILLIUM  supplier  evaluations  and  assessments,  stated  that  most 
supplier  organizations  have  reuse  libraries.  Sometimes,  however,  they  do 
not  adhere  to  a  process  that  tracks  and  maintains  the  library's  components. 
Without  this  process,  even  more  uncontrolled  cloning  can  result. 

THE  IMPACT  OF  TRILLIUM  AND  ITS  REUSE  CRITERIA  IS  PROMISING 


Francois  Coallier,  Bell's  Associate  Director  of  Quality  Engineering  and 
Research,  stated  that  while  it  is  too  early  to  measure  conclusively, 
participants  in  initial  pilots  appear  to  be  more  user-focused,  responsive  to 
customer  needs,  and  products  are  generally  more  stable.  NT  and  BNR 
managers  stated  that  they  view  TRILLIUM  as  a  means  to  achieving 
aggressive  corporate  goals.  This  enthusiasm  is  supported  by  the  fact  that 
NT  has  trained  over  90  in-house,  customer  and  supplier  staff,  and  that  tri¬ 
corporate  managers  and  practitioners  at  all  levels  reportedly  support  the 
program. 


TRILLIUM  use  is  also  expanding  to  tri-corporate  external  suppliers, 
■tovernment  organizations  and  international  standards  groups.  For 
example,  TRILLIUM  has  been  adopted  by  the  French  Ministry  of  Defense, 
and  is  strongly  shaping  the  new  International  Standards  Organization 
SPICE  (Software  Process  Improvement  and  Capability  dEtermination) 
standard. 


BOTH  SIMILARITIES  AND  DIFFERENCES  BETWEEN  BELL  AND  DOD  PROVIDE 
PERSPECTIVE. 

Bell  and  DoD  require  large,  complex  systems  with  similar  reliability 
standards  and  real-time  response  requirements.  Like  Bell's  systems,  many 
new  DoD  systems  are  written  in  a  common,  modular,  high-order 
language.  In  the  telecommunications  industry  and  in  defense  systems, 
many  products  have  well-defined  domains  and  long  life  cycles.  These 
characteristics  promote  effective  reuse  practices  such  as  those  found  in 
TRILLIUM.  Further,  Bell  and  DoD  suppliers  are,  to  a  degree,  similarly 
motivated  by  the  predominant  influence  of  their  largest  customers. 

Despite  the  influence  that  they  can  exercise  over  their  suppliers,  there  are 
technical,  regulatory,  and  organizational  differences  between  Bell  and 
DoD.  For  example,  the  relationships  between  tri-corporate  organizations 
(customers  and  suppliers)  are  much  closer  than  those  between  DoD  and 
its  contractors,  which  tend  to  be  more  formal.  Where  suppliers  often  build 
one-of-a-kind  systems  for  DoD,  Bell  supplier  NT  builds  systems  to  Bell's 
requirements,  then  resells  them  to  thousands  of  customers  worldwide. 
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DoD  has  identified  software  reuse  as  a  central  strategy  for  improving  the 
quality,  timeliness  and  cost  efficiency  of  its  weapons,  command  and 
control  and  other  automated  information  systems.  However,  to  reap  full 
benefits  from  software  reuse  technology,  DoD  must  change  the  way  it 
does  business.  The  question  is  how. 

Experience  has  shown  that  simply  asking  for  reuse  in  a  DoD  Request  for 
Proposal  (RFP)  can  result  in  more  documentation,  higher  costs  and  no 
benefits.  Wrong  policy  or  direction  can  work  against  contracts  that  try  to 
save  DoD  money  or  increase  product  quality  or  timeliness.  Emphasizing 
software  reuse  alone,  without  supporting  technology  and  mature 
processes,  can  produce  many  lines  of  useless  or  even  hazardous  software. 

This  study  provides  a  commercial  example  in  which  software  reuse  is 
specifically  gauged  and  encouraged  as  a  means  to  improve  quality, 
timeliness  and  life-cycle  cost.  TRILLIUM’s  recommended  software  reuse 
practices  are  based  upon  over  ten  years  of  successful  experience.  Reuse  is 
not  isolated,  but  viewed  as  one  of  several  activities,  all  of  which  are 
needed  to  ensure  software  capability.  Further,  evidence  suggests  that 
TRILLIUM  has  been  widely  adopted  by  managers  and  practitioners 
throughout  Bell's  supplier  organizations  (who  have  chosen  to  market 
TRILLIUM  to  their  customers).  As  such,  this  example  appears  to  provide 
the  right  goals,  long-term  experience  base  and  successful  approach  for 
getting  effective  software  reuse  into  practice. 

While  there  are  clear  differences  between  DoD  and  Bell,  we  believe  that 
the  experiences  and  lessons  learned  by  Bell  and  its  suppliers  can  provide 
perspective  and  insights  to  those  who  develop  and  implement  DoD 
policy.  The  TRILLIUM  case  study  aims  to  convey  these  points. 

For  more  information  on  the  case  study,  contact  David  M.  Dikel,  Principal 
Author,  at  Applied  Expertise,  Inc.,  1925  N.  Lynn  Street,  Suite  802,  Arlington, 
VA,  22209,  (703)  516-0911. 
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Notes: 

The  TRILLIUM  case  study  provides  a  commercial  example  of  a  software  supplier 
capability  assessment  model  that  encourages  effective  reuse.  The  study  is  structured  to 
introduce  the  reader  to  the  purpose  and  objectives  of  the  study;  define  the  scope  and 
methodology  used  by  case  study  evaluators;  provide  background  on  Bell  Canada  (Bell) 
and  TRILLIUM;  describe  the  study  results;  and  offer  our  conclusions.  The  study  is 
presented  in  the  form  of  an  annotated  briefing.  It  was  prepared  as  a  research  product 
by  Applied  Expertise,  Inc.,  in  accordance  with  IBM  and  DoD  requirements.  For  further 
information  on  the  study,  contact  David  M.  Dikel,  principal  author,  at  (703)  516-0911. 
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Problem  Statement 


DoD  sees  software  reuse  as  a  solution.  Acquisition-related  barriers  stand  in  the  way. 

Issues  include: 

•  Ensuring  that  suppliers  apply  reuse  to  achieve  DoD  objectives,  e.g.,  reduce  cost 

•  Evaluating  supplier  capability  to  deliver  on  proposed  reuse  objectives 

•  Avoiding  non-essential  or  counterproductive  requirements,  policies,  regulations, 
procedures 

•  Achieving  "grass  roots"  support  from  the  acquisition  and  vendor  communities 


Slides 

NOTES: 

The  Department  of  Defense  (DoD)  estimates  that  expenditures  for  developing  and 
maintaining  software  for  its  weapons,  command  and  control,  and  other  automated 
information  systems  currently  exceed  $24  billion  dollars  a  year.  In  an  attempt  to  better 
manage  these  costs  and  improve  its  ability  to  develop  and  maintain  high-quality  software. 
Defense  has  initiated  a  comprehensive  effort  to  incorporate  software  reuse  practices  into 
its  software  development  efforts. 

—  U.S.  GAO,  "Software  Reuse:  Major  Issues  Need  To  Be  Resolved  Before  Benefits 
Can  Be  Achieved"  i 

United  States  Department  of  Defense  (DoD)  efforts  to  encourage  effective  software 
reused  from  its  suppliers  have  met  with  a  number  of  acquisition-related  barriers.  Issues 
that  need  to  be  resolved  to  alleviate  these  barriers  include^: 

•  ensuring  that  suppliers  will  apply  reuse  to  reduce  cost,  increase  quality  and 
timeliness 

•  evaluating  supplier  capability  to  effectively  apply  reuse  technology  and 
deliver  on  proposed  reuse  objectives 

•  avoiding  the  use  of  specifications  that  impose  non-essential  or 
counterproductive  requirements,  policies,  regulations  and  procedures  on 
customers  and  suppliers 

•  achieving  "grass  roots"  support  from  the  acquisition  and  vendor  communities 


^  U.S.  General  Accounting  Office,  "Software  Reuse:  Major  Issues  Need  to  Be  Resolved  Before  Benefits  Can  Be 
Achieved,"  January  1993,  p.  1. 

2  The  DoD  defines  reuse  as,  "The  application  of  a  reusable  [software]  component  to  more  than  one  application 
system."  (Department  of  Defense,  "Software  Reuse  Initiative:  Vision  and  Strategy,"  July  15, 1992, 1st  edition, 
p.  2.)  Reuse  can  occur  within  a  system,  such  as  within  a  robot  arm;  across  similar  systems,  like  a  product  line  of 
household  appliances;  or  in  common  services  for  widely  different  systems,  for  example,  user  interface  tools. 
Any  product  created  during  the  software  development  life-cycle  is  considered  a  component,  and  a  potential 
resource  for  reuse. 

The  case  study  does  not  seek  to  address  all  of  the  acquisition-related  barriers,  such  as  data  rights  issues,  for 
example.  In  this  regard,  this  list  is  not  intended  to  be  totally  inclusive.  The  list  includes  issues  raised  during 
joint  meetings  of  the  DoD  Management  Issues  Working  Group  and  ACM/SIGAda  Reuse  Acquisition  Action 
Team,  as  well  as  during  interviews  with  our  case  study  review  team,  a  group  that  represents  the  concerns  of 
DoD  and  NASA  policymakers.  Program  Executive  Officers,  Program  Managers  and  senior  technical  managers 
from  the  defense/aerospace  community. 
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STARS-funded  Approach 


The  TRILLIUM  Case  Study  looks  at  getting  effective  reuse  into  practice 

•  From  the  perspective  of  the  customer 

•  As  part  of  an  overall  quality  initiative 

Slide  4 

Notes: 

This  illustrative  case  study  was  prepared  for  DoD’s  Advanced  Research  Project 
Agency  (ARP A)  Software  Technology  for  Adaptable  Reliable  Systems  (STARS)  program 
to  provide  insights  and  suggest  strategies  towards  the  resolution  of  these  issues.  The 
study  was  authorized  under  a  sub-contract  to  IBM  Federal  Systems  Company.  The 
study  focused  on  the  joint  effort  of  Bell  Canada,  Northern  Telecom  and  BeU  Northern 
Research  to  produce  a  software  product  development  capability  evaluation  mod^*l 
called  TRILLIUM.  The  aim  of  the  case  study  is  to  examine  Bell  Canada's  efforts  to 
promote  effective  supplier  participation  in  key  technologies  —  in  this  case,  software 
reuse  —  to  increase  their  software  capability  in  the  context  of  an  overall  quality 
initiative.  TRILLIUM  is  a  self-assessment  tool  which  includes  software  reuse  as  a 
significant  evaluative  factor.  It  allows  continual  improvement  of  software  processes 
through  self-assessment. 
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Why  r/?/LL/t/Af?  (lof2) 


Bell  Canada  system  requirements  resemble  those  of  DoD 

End-to-end  service  depends  on  software  systems  that  are  large,  critical,  complex  and 

subject  to  rapid  changes 

Digital  switch  characteristics 

•  System  size:  >10  MLOC  (million  lines  of  code) 

•  Almost  every  installation  has  a  different  feature  set 

•>2,400  installations  worldwide 

•  Continuous  demand  for  new  features: 

-  Typical  update 

-  has  400  features,  each  with  about  5,000  LOC 

-  represents  efforts  of  1,200  s/w  designers 

•  Updates  must  not  impact  availability 

<  2  hours  downtime  in  40  years 

Do  not  use  these  data  to  calculate  productivity 

Slides 

NOTES: 

Northern  Telecom's  DMS  series,  which  is  a  product  line  of  digital  switching  systems, 
performs  a  broad  range  of  telecommunications  applications,  including  toll,  local  and 
international  services  in  over  2,400  installations  worldwide  (1989).  Each  installation 
requires  a  unique  combination  of  product  features. 

NT  develops  new  features  for  regularly  scheduled  release  to  meet  aggressive 
customer  demand.  A  typical  release  of  software  has  400  features,  each  using  an  average 
of  5,000  lines  of  code.  Each  release  represents  the  collective  efforts  of  some  1,200 
software  designers  located  in  the  U.S.,  Canada  and  the  U.K.  Releases  must  be  installed 
quickly,  while  maintaining  system  reliability  requirements  of  less  than  two  hours 
downtime  over  a  40  year  period  of  operation. 
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Why  r/g/LL/t/A/?  (2  of  2) 


Bell  incorporated  siw  reuse  practices  into  a  framework  that 


•  Stresses  improving  supplier  responsiveness,  increasing  quality  and  reducing  cost 

•  Enjoys  the  committed  support  of  senior  managers  within  Bell  Canada,  Northern 
Telecom  and  Bell  Northern  Research 

•  Involves  participation  from  ail  practitioners  and  managers  (customer  and 
supplier) 

•  Is  central  to  a  Total  Quality  Management  (TQM)  program 

Slide  6 

Notes: 

The  TRILLIUM  model  expands  the  Software  Engineering  Institute’s  Capability 
Maturity  Model  (SEI/CMM)  to  include  a  strong  customer  focus,  a  product  perspective, 
technological  maturity  and  software  reuse.  The  model  consists  of  a  series  of  levels 
which  represent  an  organization's  software  development  maturity.  At  each  level  there 
are  key  work  practices  which  must  be  enacted  to  advance  to  the  next  level.  TRILLIUM 
incorporates  quality  management  practices,  a  number  of  industry  standards  as 
benchmarks  for  assessments,  and  specific  customer  requirements  for  each  assessment.*^ 
Software  reuse  practices  exist  at  levels  two  and  beyond  (for  a  detailed  outline  of  these 
practices,  see  the  index  to  TRILLIUM,  attached  as  appendix  V).  Most  of  these  reuse 
practices  are  original  to  TRILLIUM  and  are  not  based  on  the  CMM,  IEEE  standards,  or 
other  inputs  to  the  model. 

TRILLIUM  supports  an  assessment  methodology  and  a  questionnaire  that  can  be 
used  jointly  by  customer  and  supplier  in  self-assessment  mode,  or  alternately  by  the 
customer  in  audit  mode.  TRILLIUM  self-assessments  enable  the  supplier  to  openly 
identify  areas  that  need  improvement  in  the  presence  of  the  customer;  as  such,  self- 


4  The  authors  of  the  model  stressed  that  TRILLIUM  standards  and  evaluation  criteria  are  not  immutable.  They 
are  provided  as  a  framework  to  assess  the  software  development  process  and  are  meant  to  encourage  continuous 
improvement 
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assessments  require  and  hopefully  foster  a  significant  level  of  trust  between  customer 
and  supplier.  According  to  Bell  managers,  self-assessments  can  provide  customers 
strong  assurance  that  suppliers  are:  (i)  committed  to  continuous  quality  improvements, 
(ii)  aware  of  key  areas  that  need  improvement,  and  (iii)  taking  aggressive  and  specific 
actions  to  improve.  In  assessment  mode,  a  team  of  evaluators  consists  of  members  of 
the  supplier  organization  and  at  least  one  member  of  the  customer  organization.  The 
team  uses  the  TRILLIUM  questionnaire  and  interviews  project  managers  and  product 
users  to  identify  weaknesses  in  the  development  process.  These  weaknesses  are  then 
addressed  in  an  effort  to  increase  product  quality  and  reduce  costs. 

In  audit  mode,  customers  use  the  model  as  a  standard  by  which  to  choose  among 
potential  suppliers,  or  as  a  tool  to  analyze  and  remedy  serious  cost/schedule/quality 
problems  with  ciorrent  suppliers.  Bell  chooses  between  these  options  based  on  supplier 
performance  and  the  criticality  of  supplier  software  products.  TRILLIUM  incorporates 
national  and  international  standards  into  its  assessments. 
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Purpose  and  Objectives 


•  Determine  the  process  and  rationale  for  TRILLIUM  reuse  criteria 

•  Gauge  internal  and  external  support  for  these  criteria 

•  Discern  the  impact  of  TRILLIUM  and  its  reuse  practices 

•  Describe  similarities  and  differences  between  Bell  Canada  and  DoD 

Slide? 

NOTES: 

The  TRILLIUM  case  study  was  prepared  for  DoD  officials  to  provide  insights  and 
suggest  strategies  towards  breaking  down  acquisition-related  barriers  to  effective 
software  reuse. 

The  objectives  of  the  case  study  were  to: 

•  Determine  the  process  and  rationale  that  led  Bell  Canada,  Northern  Telecom, 
and  Bell  Northern  Research  to  require  and  include  reuse  in  their  criteria  for 
evaluating  software  capability. 

•  Gauge  the  level  of  internal  and  external  support  for  including  reuse  as  a 
critical  criterion  in  Bell  Canada’s  software  supplier  quality  evaluations. 

•  Discern  the  impact  of  TRILLIUM  and  its  associated  reuse  practices. 

•  Describe  similarities  and  differences  between  Bell  Canada  and  DoD  in 
purchasing  and  developing  software  products. 
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Effort  -  about  three  person  months  including 

•  Interviews  with  Case  Study  Review  Team 

•  Site  visits  to  Bell,  BNR  and  Ecole  Polytechnique 

•  Additional  face-to-face  interviews  at  IEEE  /  Quality  Management  Assurance 
Committee  (QMAC)  Workshop 

•  Collection  and  review  of  documents  from  tricorp  managers,  industry  journals, 
annual  reports,  SEI,  etc. 

Coverage 

•  3  of  3  TRILLIUM  principal  authors 

•  2  of  6  initial  TRILLIUM  implementors 

•  BNR  Software  Reuse  Program  Manager 

•  Corroborating  interviews  at  several  levels  of  each  tri-corporate  organization 

Caveats 

•  Most  subjects  had  a  stake  in  TRILLIUM 

•  Do  not  use  data  to  estimate  productivity  —  counts  of  lines  of  code/programmers 
are  provided  for  comparisons  only 

-  BNR  mgr.  noted  that  the  #  of  programmers  can  vary  by  an  order  of 
magnitude,  "depending  on  how  you  count" 

Slides 

Notes: 

Field  work  for  the  case  study  was  accomplished  during  the  period  May  through  July 
of  1993.  A  fact  sheet  was  provided  to  tri-corporate  managers  for  comments.  In  addition, 
an  annotated  briefing  was  held  for  selected  reviewers  of  the  case  study  in  August  1993 
(see  Appendix  II  for  list  and  titles  of  reviewers).  Where  appropriate  and  necessary, 
comments  received  from  tri-corporate  managers  and  reviewers  were  considered  in  the 
preparation  of  the  final  case  study. 

We  reviewed  and  analyzed  documents  and  materials  related  to  the  development 
and  application  of  TRILLIUM-,  held  interviews  in  Montreal  and  Ottawa  with  managers 
and  technologists  of  Bell  Canada,  Northern  Telecom  and  Bell  Northern  Research;  and 
interviewed  the  Program  Manager  for  industry  software  development  projects  at  Ecole 
Poly  technique  in  Montreal,  whose  group  develops  and  supplies  case  tools  to  Bell.  In 
order  to  develop  a  broader  perspective  on  TRILLIUM ,  information  was  obtained  from 
different  levels  (customer  and  supplier)  of  the  tri-corporate  structure.  We  also 
conducted  face-to-face  interviews  at  a  telecommunications  quality  management 
workshop  (the  lEEE/QMAC  workshop),  where  we  gathered  background  information 
from  Bell  suppliers  and  heard  presentations  on  uses  of  capability  maturity  models 
within  the  telecommunications  industry. 
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The  information  used  in  the  preparation  of  the  case  study  was  acquired  primarily 
from  individuals  directly  responsible  for  the  development  and  implementation  of 
TRILLIUM.  We  spoke  with  each  of  TRILLIUM'S  three  principal  authors,  two  out  of  six 
of  its  initial  implementors,  and  the  BNR  Software  Reuse  Program  Manager.  We  also 
conducted  corroborating  interviews  at  several  levels  of  each  tri-corporate  organization. 

These  individuals  have  an  investment  and  vested  interest  in  the  success  of 
TRILLIUM  within  the  tri-corporate  structure  and  as  part  of  their  commitment  to  quality 
management.  Nevertheless,  the  corroboration  of  information  from  different  corporate 
levels  and  elements  of  the  tri-corporate  structure  provides  reasonable  assurance  that  the 
information  obtained  is  reliable.  In  this  regard,  to  the  extent  practical,  the  case  study 
attributes  the  information  to  its  source. 
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•  Tri-corporate  Relationships 

•  Quality,  TRILUUM  and  Reuse 

•  TRILUUM  History  and  Implementation 

Slide  9 

Notes: 

The  Background  section  is  intended  to  provide  perspective  to  the  reader  by 
describing  customer-supplier  relationships,  their  commitment  to  quality,  and  the 
history  and  implementation  of  TRILLIUM. 
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Tri-corporate  Relationships 


SUPPLIERS  CUSTOMERS 


*  Ecole  supplies  signifiesnUy  less  than  the  other  organaations  listed 

Bell,  NT  and  BNR  all  have  software  suppliers  that  are  not  reflected  in  this  diagram 
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Notes; 

The  tri-corporate  family  consists  of  Bell  Canada  (Bell),  Northern  Telecom  (NT)  and 
Bell  Northern  Research  (BNR),  which  are  owned  wholly  or  in  part  by  the  parent  holding 
company,  BCE,  Inc.^.  Bell  is  the  largest  Canadian  supplier  of  telecommunications 
services,  securing  approximately  sixty  percent  of  the  Canadian  market.  It  provides 
voice,  data  and  image  services,  and  the  management  and  operating  systems  required  to 
sustain  them.  These  services  are  based  on  technologies  primarily  provided  by  NT  and 
BNR. 

NT's  digital  switches  are  at  the  heart  of  Bell’s  systems,  and  NT  supplies  more  than 
two  thirds  of  Bell's  total  software  purchases.  NT,  in  turn,  relies  on  BNR,  its  R&D 
subsidiary,  for  the  software  required  to  develop  its  switching  and  transmissions 
systems.®  BNR's  pioneering  innovations  in  digital  telecommunications  systems  has 
made  NT  the  world's  largest  supplier  of  digital  switches.  According  to  A1  Graydon, 
Manager  of  Software  Excellence  at  NT,  in  1985  the  company  had  less  than  $US  1  billion. 
In  1992  it  posted  sales  in  excess  of  $8.4  billion.  This  growth  was  mainly  due  to  the 
competitive  advantage  from  implementing  digital  signal  processing  technologies  before 
competitors.  Quality  initiatives  provided  through  TRILLIUM  are  viewed  as  key  to 
sustaining  growth. 


®  BCE,  Inc.  owns  100%  of  Bell  52.7%  of  NT,  and  100%  of  BNR  through  Bell  and  NT.  Bell  owns  a  30%  share  of 
BNR,  and  NT  is  the  majority  owner  with  70%. 

®  BNR  produces  99%  of  NTs  software  and  receives  90%  to  95%  of  their  funding  from  NT. 
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uality,  TRILLIUM  and  Reuse 
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NOTES: 


Interviewees  stressed  that  effective  software  reuse  cannot  be  achieved  in  isolation. 
TRILLIUM  places  reuse  as  one  of  several  important  development  practices.  In  this 
context,  reuse  success  is  tied  to  an  organization’s  overall  software  capability  and  its 
commitment  to  continuous  quality  improvement. 

The  TRILLIUM  model  defines  capability  in  this  context  as,  "The  ability  of  a 
development  organization  to  deliver  a  sohware  product  that  meets  customer 
expectations  with  minimal  defects,  for  the  lowest  life-cycle  cost,  in  the  shortest  calendar 
time."^  The  TRILLIUM  model  is  comprised  of  nine  capability  areas,  ranging  from 
quality  management  to  customer  support  to  development  practices.  Included  within 
seven  development  practices  are:  configuration  management  (CM),  verification  and 
validation  (V&V),  reliability  management,  and  software  reuse  (see  Appendix  V). 


^  Bell  Canada  Document,  "Trillium:  Telecom  Software  Product  Development  Capability  Assessment  Model," 
July  1992,  Draft  2.2,  p.  2. 
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TRILLIUM  Histor 


•  Foundation  document  presented  to  ASQC  Conference  by  Francois  Coallier  [July 
•91] 

•  Tri-corporate  team  formed  [October  *91] 

•  Five  self-assessments  conducted  [April  '92  -  April  *93] 

•  Fiber  Systems  self-assessment  [April  *92] 

-  Fiber  Systems  builds  four  products  requiring 

•  >3  MLOC  each 

•  about  2,000  programmers 

•  >15  assessments  and  audits  of  Bell's  external  suppliers 
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NOTES: 

TRILLIUM  was  developed  from  a  tri-corporate  joint  effort  to  assess  and  enhance 
NT's  and  BNR's  software  development  processes.  After  conducting  manufacturing 
audits  for  many  years.  Bell  decided  that  it  was  necessary  to  analyze  supplier  software 
development.  Around  1982,  Bell  began  auditing  NT's  software  using  IEEE  standards 
[e.g.  IEEE  730].  By  1988  these  audits  started  to  look  at  supplier  capability  using  a  simple 
questionnaire  in  conjunction  with  the  IEEE  standards.  This  change  in  approach  resulted 
from  concerns  with  supplier  performance,  life-cycle  costing  and  product  reliability  in 
light  of  the  rapid  growth  in  system  size  and  complexity.  At  this  point.  Bell  also  began  to 
fund  academic  and  BNR  research  into  solutions  to  these  issues. 

These  audits  were  concomitant  with  NT's  introduction  of  "Excellence,"  a  TQM 
program  which  would  play  a  key  role  in  TRILLIUM  implementation.  This  program 
quickly  moved  into  BNR  and  was  in  full  gear  in  both  organizations  by  1990.  Further 
setting  the  stage  for  TRILLIUM  was  the  1989  consolidation  of  responsibilities  under  one 
director  for  all  aspects  of  quality  (e.g.,  hardware  and  software).  This  increased  the 
leverage  of  the  software  quality  initiative,  since  it  represented  a  high  priority  for  Bell's 
continuous  quality  assurance  process. 

In  1990,  Bell  began  to  look  at  a  comprehensive  approach  to  ensuring  quality  in  the 
software  development  process.  A1  Graydon,  NT's  Manager  of  Software  Excellence, 
stated  that  at  this  time.  Bell  approached  BNR  and  NT  with  the  prospect  of  expanding 
the  scope  of  software  audits  to  the  level  of  detail  found  in  manufacturing.  NT  and  BNR 
felt,  however,  that  traditional  audits  were  not  the  best  method  of  approaching  the  task. 
A  tri-corporate  team  was  formed  to  look  into  other  evaluation  methods.  The  fourteen- 
member  tri-corporate  team  was  headed  by  Neil  Gammage,  then  Program  Director  for 
the  Software  Engineering  Centre  at  BNR,  Francois  Coallier,  Associate  Director  of 
Quality  Engineering  and  Research  at  Bell,  and  A1  Graydon.  The  objective  was  to  create 
a  model  which  would  take  the  best  of  existing  evaluation  methods  and  combine  them 
into  a  unified  whole. 
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TRILLIUM  was  first  tested  as  an  evaluative  tool  at  BNR  in  the  summer  of  1991.  The 
first  self-assessment,  led  by  Don  Joyce,  Manager  of  Transmission  Development 
Processes,  was  conducted  at  Fiber  Systems  in  1992.  Fiber  Systems  now  builds  several 
systems  that  contain  over  3  MLOC  each  and  involve  about  2,000  programmers, 
according  to  Joyce.  Joyce  was  introduced  to  what  became  TRILLIUM  in  late  1991  and 
helped  to  promote  the  methodology  to  BNR  upper  management  and  throughout  the 
organization.  The  first  draft  of  what  became  TRILLIUM  was  published  in  July  of  1991 
and  was  presented  by  Coallier  at  the  First  Annual  American  Society  for  Quality  Control 
(ASQC)  Software  Conference  in  October  1991. 

Since  the  first  BNR  self-assessment,  TRILLIUM  has  spread  throughout  NT  and  its 
supplier  base.  To  date,  five  full  assessments  and  numerous  Bell  audits  have  been 
completed.  TRILLIUM  has  evolved  to  reflect  the  acquired  knowledge  from  the 
assessments  and  has  been  fully  integrated  into  the  overall  quality  improvement  efforts 
within  the  tri-corporate  organizations. 
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•  TRILUUM  criteria  were  intended  to  improve  and  expand  on  existing  Tri¬ 
corporate  software  reuse  practices 

•  TRILLIUM  and  its  reuse  criteria  enjoy  strong  support  in  Tri-corporate 
organizations 

•  The  impact  of  TRILLIUM  and  its  software  reuse  criteria  looks  promising 

•  Similarities  and  differences  with  DoD  provide  perspective 
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NOTES; 

During  our  research,  we  found  several  examples  that  we  believe  demonstrate 
competitive  advantage  gained  from  effective  software  reuse.  One  such  example,  BNR 
switching  systems,  is  discussed  in  this  study.  Switching  systems  provided  a  long-term 
experience  base  upon  which  TRILLIUM'S  reuse  criteria  were  formed.  We  found 
widespread  support  for  these  practices  and  some  evidence  of  impact.  We  also 
identiHed  similarities  and  differences  between  Bell  and  DoD  which  need  to  be 
considered.  The  following  annotated  briefing  charts  describe  the  results  of  our  study. 
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TRILLIUM  Criteria  Intended  to  Improve  and 
Expand  on  Existing  Reuse  Practices  (1  of  4) 


Bell  System  Characteristics  and  Competitive  Pressures  Engendered  Reuse 

•  Updates:  2  MLOC  every  six  months 

•  Updates  must  not  impact  availability 

•  Hundreds  of  configurations  —  almost  every  installation  has  a  different  set  of 
features 

For  Bell,  new  features  mean  increased  revenues  and  staying  ahead  of  competition 
For  Northern  Telecom,  new  features  mean  market  share 
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NOTES: 

TRILLIUM  Criteria  Sought  To  Improve  and  Expand  On  Existing  Tri-corporate 
Reuse  Practices 

Reuse  has  long  been  a  part  of  software  development  in  tri-corporate  organizations. 
For  example,  over  fifteen  years  ago.  Bell  suppliers,  NT  and  BNR,  created  a  product 
architecture  and  reuse  library  to  develop  quality  products  within  reasonable  cost 
constraints  for  digital  switching  systems.  For  other  products,  however,  reuse  practices 
were  not  nearly  as  advanced.  Design  architecture  was  uneven  throughout  tri-corporate 
systems  and  some  reuse  took  the  risky  form  of  unmanaged  cloning.® 

In  an  effort  to  address  this  situation,  tri-corporate  managers  looked  at  successful 
systems  which,  because  of  their  size  and  complexity,  stable  well-defined  domains  and 
long  life  cycles,  had  engendered  effective  reuse  practices.  These  systems  were  used  to 
formulate  a  set  of  "best  practices,"  many  of  which  were  incorporated  into  TRILLIUM  to 
provide  structure  and  management  to  existing  software  reuse  practices,  and  to  expand 
the  scope  of  reuse  throughout  tri-corporate  systems. 

Bell  System  Characteristics  and  Competitive  Pressures  Engendered  Reuse 

NT/BNR  digital  switching  systems  provide  an  example  of  systems  that  have 
benefited  from  reuse.  Reuse  practices  were  developed  to  meet  customer  demands  for 
continuous  system  upgrades.  Reuse  was  feasible  because  system  characteristics  enabled 
developers  to  recover  the  added  initial  costs  associated  with  these  practices. 

DMS  systems  perform  a  broad  range  of  telecommunications  applications,  including 
toll,  local  and  international  services  in  over  two  thousand  four  hundred  (2,400) 
installations  worldwide  (1989). 

Each  installation  requires  a  unique  combination  of  product  features. 

NT  develops  new  features  for  release  every  six  months  to  meet  aggressive  customer 
demand.  A  typical  release  of  software  has  four  hundred  features,  each  using  an  average 
of  five  thousand  (5,000)  lines  of  code.  That  is,  two  million  (2M)  lines  of  new  code  are 


®  Interviewee  definitions  of  cloning  ranged  from  copying  a  major  software  product  and  modifying  one  or  more 
modules,  to  copying  a  small  module  with  little  or  no  modification. 
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produced  and  implemented  every  six  months.  Each  release  represents  the  collective 
efforts  of  some  twelve  htmdred  (1,200)  software  designers  located  in  the  U.S.,  Canada 
and  the  U.K.  Releases  must  be  installed  quickly  while  achieving  system  reliability 
requirements  which  limit  downtime  to  less  than  two  hours  over  a  forty  year  period  of 
operation. 

Reuse  was  the  natural  reaction  to  these  stringent  requirements.  As  stated  by  Neil 
Gammage,  Director  of  Synergy  Architecture  at  BNR; 

"The  only  way  to  make  order  of  magnitude  improvements  in  software  quality  and 
developer  productivity  is  to  write  less  code  —  to  reuse. " 

NT  and  BNR  reuse  practices  evolved  to  meet  the  challenges  inherent  in  developing 
these  systems. 
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TRILLIUM  Criteria  Intended  to  Improve  and 
Expand  on  Existing  Reuse  Practices  (2  of  4) 


Reuse  Involved  Architecture,  Processes  and  Tools 

•  Product-line  architecture  went  virtually  without  change  for  15  years 

•  Distributed  Product  Library 

-  controls  over  100  libraries  of  software,  firmware,  hardware  and 
documentation  and  enforces  process 

-  supports  fifteen  different  releases 

-  entire  product  line  generated  >20  MLOC 

•  Product  Tracking  System  tracks  individual  features,  test  cases  and  problem 
reports 

•  High  level  programming  language  uniformity  supports  modularity  and 
information  hiding 
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NOTES: 

Reuse  Involved  Architecture.  Processes  and  Tcx)ls 

To  manage  and  control  the  systems  described  above,  BNR  developed  product 
architectures,  a  set  of  sophisticated  tools,  and  a  carefully  controlled  process.  Modular 
architecture  for  both  hardware  and  software  was  developed  to  allow  efficient  expansion 
and  upgrading  of  individual  systems  in  excess  of  ten  million  (lOM)  lines  of  code, 
without  disturbing  system  operations.  A  central  library  evolved,  from  which  all 
systems  are  currently  generated.  This  resource  presently  contains  more  than  twenty 
million  (20M)  lines  of  source  code. 

Because  switching  systems  architecture  was  well  designed  in  the  initial  stages,  it  has 
needed  only  minor  Ganges  to  support  changes  in  technology  during  its  initial  twelve 
years.  As  a  result,  NT  has  been  able  to  evolve  its  existing  switch  base  rather  than 
replace  it.  This  architecture  enabled  NT  to  offer  customers  the  ability  to  select  only 
features  they  required,  while  its  competition  offered  only  a  few  variations.  This 
"evergreen"  capability  has  been  a  key  factor  in  NT's  success  with  these  systems,  and  in 
turn,  the  telecommunications  services  provided  by  Bell.  Recently,  however,  the 
requirements  of  new  technology  and  expanded  customer  demands  have  outstripped 
the  capability  of  system  architecture  and  tools  to  respond.  NT  is  now  in  the  process  of 
implementing  several  aggressive  initiatives  designed  to  meet  these  demands.  Software 
reuse  is  a  key  element  in  this  new  approach.  (See  following  section  for  more  detail.) 

Tools  and  processes  helped  to  manage  the  development  environment  and  control 
reuse.  In  1989  and  1990,  BNR's  Telesis  publication  described  a  multi-site  integrated 
"Product  Development  Environment,"  consisting  of  a  "Product  Library  System"  and  a 
"Product  Tracking  System."  According  to  the  publication,  the  Product  Library  System 
organizes  and  manages  over  one  hundred  libraries  of  software,  firmware,  hardware  and 
documentation.  It  stores  and  manages  the  product’s  source  files,  including  code, 
documentation,  and  data.  The  library  system  can  simultaneously  support  fifteen 
different  releases  of  software.  The  Product  Tracking  System  tracks  the  progress  of 
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individual  features,  the  development  and  execution  of  test  cases,  and  the  resolution  of 
internal  and  external  problem  reports.  According  to  Telesis,  the  environment  "ensures  a 
standard  design  process  and  delivery  methodology  for  DMS  software  across  several 
sites  and  design  groups."^ 

Further,  the  Product  Development  Environment’s  storage  and  reporting  capability 
"enables  the  BNR  design  community  and  Northern  Telecom  to  know  exactly  what  is  in 
each  software  release  at  any  time,  liie  environment  can  instantly  tell  them,  for  example, 
what  features  have  been  developed,  what  problems  fixed,  the  number  of  test  cases 
executed,  and  the  pass  rates  on  test  cases  and  problem  fixes."^® 

Reuse  was  further  promoted  by  language  uniformity  in  software  development 
projects.  Virtually  all  tri-corporate  software  is  written  in  a  single,  proprietary,  high- 
level  programming  language  called  PROTEL,  which  supports  programming  modularity 
and  information  hiding. 

For  further  information  please  consult  the  following  sources: 

Johns,  Kalli,  "Product  Development  Environment:  Managing  large  software  systems," 
Telesis  1989  three,  BNR,  pp.  51-58. 

Cashin,  Peter,  "BNR  remains  at  forefront  of  computing  technology,"  Telesis  issue  no.  92, 
pp.  73-75. 

Stewart,  Ian,  "From  POTS  to  intelligent  services,"  Telesis  issue  no.  92,  pp.  94-98. 


^  Johns,  Kalli,  "Product  Development  Environment;  Managing  large  software  systems,”  Telesis  1989  three,  BNR, 
pp.  53. 

10  Ibid. 
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Architecture  and  Environment  Maturity  is  a 
Double-edged  Sword 


•  Innovative  architecture,  process  and  support  system  allowed  NT  to  expand  into 
new  markets  quickly 

•  Total  control  of  development/target  environment  (compiler,  tools, 

0/S  and  hardware) 

•  Modular  architecture  allowed  rapid  tailoring  of  product 
-  Approach  resulted  in  exponential  code  growth 

•  Several  years  ago  requirements  stripped  capacity 

•  Lack  of  ability  to  leverage  advances  in  COTS  technology 

•  New  environment  and  approach  proposed  —  senior  management  balks  at  risk 

•  Result:  Recently  announced  problems 

New  architecture,  case  tools,  object-oriented  technologies  and  software  reuse,  are  key 

elements  of  NT's  solution 


Slide  16 

NOTES: 

BNR's  innovative  architecture,  process  and  support  system  allowed  NT  to  quickly 
expand  into  new  markets.  Complete  control  over  development  and  target 
environments,  as  well  as  a  modular  architecture,  permitted  NT  to  develop  more  new 
features  faster  than  the  competition,  to  respond  to  thousands  of  unique  combinations  of 
customer  requirements,  and  to  adapt  to  rapid  changes  in  technology.  This  approach 
also  resulted  in  exponential  code  growth  and  restricted  BNR’s  ability  to  leverage 
commercially  available  off-the-shelf  case  tools  and  technologies.  In  recent  years, 
however,  requirements  have  started  to  exceed  capacity.  Recently  announced  problems 
by  NT  reflect  this  observation. 

In  July  1993,  NT  announced  a  second  quarter  loss  of  $US  1.03  billion.  It  now  expects 
a  loss  for  the  full  year  1993.  According  to  Jean  C.  Monty,  president  and  chief  executive 
officer  of  NT,  among  the  major  factors  impacting  the  quarter  were  declines  in  gross 
margins,  investments  in  restructuring  and  software,  and  significantly  lower  Central 
Office  Switching  (COS)  software  sales.  According  to  a  BNR  manager,  much  of  the 
decline  in  software  sales  is  due  to  a  ten-month  slippage  in  the  latest  software  update. 
While  COS  revenues  declined,  product  line  revenues  "showed  good  growth"  in 
Transmission  and  Multimedia  Communications  Systems.’^ 

According  to  Business  Week,  the  recent  troubles  began  last  year  when  NT’s  phone 
switches  required  updating.  Several  U.S.  companies  experienced  problems  with  the 
software  in  NT’s  phone  switches;  they  complained  about  the  time  it  took  for  NT  to 
respond  to  their  demands  to  fix  the  glitches.^^  According  to  Business  Week,  late  last  year 
Monty  proposed  to  then-CEO  Paul  G.  Stern  a  major  software  revamping  and 


*  *  Northern  Telecom  news  release,  "Northern  Telecom  Announces  Second  Quarter  Results,”  July  21 , 1993. 

Ziegler,  Bart,  "What  Really  Happened  at  Northern  Telecom,"  Business  Week,  August  9, 1993,  p.  27. 
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simplification.  Stem  reportedly  agreed  in  principle,  but  the  issue,  said  Monty,  was  one 
of  "how  much  and  how  fast."  Stern  considered  the  software  revamping  but  "could 
never  figure  out  how  to  do  it"  without  disrupting  operations.  The  result  became  the 
recently  announced  problems  by  NT.^^ 

NT's  board  of  directors  has  reacted  to  current  problems  by  approving  special 
charges  for  restructuring  and  a  new  software  initiative.  "The  investments  in 
restructuring  and  software  are  expected  to  significantly  reduce  operating  costs  and 
enhance  the  software  delivery  process  to  the  benefit  of  our  customers  and  our 
shareholders,"  said  Monty.  "We  expect  to  return  to  profitability  no  later  than  fourth 
quarter  1993."  The  restructuring  program  will  cost  $US  282  million  after  tax,  and  the 
software  initiative  $US  158  million  after  tax.  The  objective  of  the  latter,  according  to 
Monty,  is  "to  generate  competitive  advantage,"  including:  "best  in  class  quality, 
significant  increases  in  design  productivity,  and  an  improved  capability  to  rapidly 
deliver  customized  products  to  individual  customers  in  diverse  global  markets."  The 
development  is  scheduled  for  completion  in  1994,  with  market  rollout  slated  for  1995.^^ 

These  problems  were  identified  several  years  ago  by  NT  and  BNR  managers.  They 
proposed  the  following  aggressive  measures  to  address  them: 

•  Improving  software  capability  (TRILLIUM) 

•  Redesigning  the  product  architecture 

•  Adopting  object-oriented  technology  and  case  tools 

•  Implementing  advanced  software  reuse  techniques 

According  to  an  NT  manager,  NT  and  BNR  took  immediate  action  to  develop  and 
adopt  a  capability  assessment  model.  This  action  resulted  in  TRILLIUM.  As  noted 
above,  additional  measures  requiring  top-level  approval  are  now  in  progress. 


13  Ibid. 

News  release,  op.  cit. 
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TRILLIUM  Criteria  Intended  to  Improve  and 
Expand  on  Existing  Reuse  Practices  (3  of  4) 


TRILLIUM  reuse  practices  were  based  on  long-term,  in-house  experience 
Level  1*  practices  are  random,  may  involve  uncontrolled  cloning 
Level  2  and  3  practices  were  chosen  to  ensure  high  return  on  investment 
•  Level  two  practices  harness  and  manage  "cloning" 

-  Level  three  practices  control,  manage  and  maintain  library  artifacts 
*  Levels  based  on  a  scale  of  1  to  5,  with  level  5  being  the  highest 
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NOTES: 

TRILLIUM  Reuse  Practices  Were  Based  On  Long-Term.  In-House  Experience 

NT  and  BNR  gleaned  valuable  information  in  developing  their  switching  systems, 
which  were  used  to  create  the  reuse  criteria  and  technology  road  map  set  by  TRILLIUM. 
Specific  TRILLIUM  reuse  practices  were  chosen  because  they  brought  the  highest  return 
on  investment,  or  as  Gammage  stated  it,  "showed  the  most  bang  for  the  buck. "  In  addition 
to  in-house  experience,  these  criteria  also  reflect  information  learned  from  state  of  the 
industry  and  state  of  the  art  practices. 

TRILLIUM  reuse  criteria  are  characterized  as  follows.  Level  one  software 
development  is  a  random  process  which  may  involve  uncontrolled  cloning;  level  two  is 
basic,  managed  cloning,  requiring  documentation  and  tracking  of  all  modules;  and  level 
three  requires  a  carefully  controlled  library  of  source  code  templates  and  units  along 
with  the  proper  organizational  structure  to  develop,  certify,  document  and  maintain  the 
items  in  the  library.  At  level  four  and  above  are  advanced  reuse  practices  that  are  only 
projections  for  most  organizations  involved  in  software  development.  Interviewees 
were  not  specific  as  to  the  extent  that  they  were  using  level  four  reuse  practices. 
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TRILLIUM  Criteria  Intended  to  Improve  and 
Expand  on  Existing  Reuse  Practices  (4  of  4) 


Reuse  Without  Appropriate  Management  Is  Inefficient  and  Risky 

•  Uncontrolled  "cloning"  can  be  hazardous 

•  Use  of  libraries  without  process  and  infrastructure  risks  systemic  software  flaws 

•  For  Bell,  effective  development  means  identifying  reuse  potential  at  the  design 
stage 

"When  [the  proper]  infrastructure  does  not  exist,  attempted  reuse  practices  show 

systemic  flaws." 

—  Don  Joyce,  Manager  of  Transmission  Development  Processes 
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Reuse  Without  Appropriate  Management  Is  Inefficient  and  Risky 

Although  recommendations  and  strategies  differed  between  interviewees,  there  was 
consensus  that  appropriate  management  of  software  reuse  at  all  stages  of  the 
development  process  is  a  very  important  element  of  a  successful  reuse  initiative.  For 
tri-corporate  managers,  appropriate  management  includes: 

•  Controlling  the  cloning  process 

•  Supporting  library  use  with  a  defined  process  and  infrastructure 

•  Identifying  reuse  potential  at  the  design  stage 

Interviewees  stated  that  architecting  products  at  the  design  stage  to  enable  reuse 
during  their  implementation  is  equally  important.  These  practices  help  organizations 
realize  the  potential  benefits  of  reuse  while  mitigating  proliferation  of  bugs  in  cloned 
modules. 

Prior  to  TRILLIUM,  tri-corporate  managers  recognized  the  prevalence  of 
"uncontrolled  cloning"  and  the  potential  hazards  of  this  practice.  One  example  that 
brought  this  to  their  attention  was  a  code  analyzer  experiment  conducted  at  Ecole 
Polytechnique.  While  studying  a  large  segment  of  Bell  code.  Bell  managers  were 
surprised  to  find  numerous  modules  with  the  same  name  and  slightly  different  code, 
and  others  with  identical  code  and  slightly  different  names.  Though  the  program  still 
worked  properly,  the  design  inefficiency  and  potential  for  serious  problems  with  bugs 
were  obvious.  TRILLIUM  authors  created  the  level  two  TRILLIUM  criteria  to  address 
the  issue.  According  to  Don  Joyce,  Manager  of  Transmission  Development  Processes, 

"When  [the  proper]  infrastructure  does  not  exist,  attempted  reuse  practices  show 

systemic  flaws. " 


Another  area  on  which  tri-corporate  managers  have  focused  is  the  appropriate 
management  of  reuse  libraries.  Jean  Normand  Drouin,  who  has  conducted  over  fifteen 
TRILLIUM  supplier  evaluations  and  assessments,  has  found  that  most  organizations 
meet  level  two  requirements  and  many  have  already  implemented  reuse  libraries. 
Sometimes,  however,  they  do  not  adhere  to  a  process  that  tracks  and  maintains  the 

September  13, 1993  Page  23 


library's  components.  Without  this  process,  even  more  uncontrolled  cloning  can  result. 
He  stressed  that  these  organizations  must  adhere  to  a  library  management  process  to 
make  reuse  successful  at  this  level.  This  includes  tracking  modules  carefully,  having 
ownership  of  modules  reside  with  one  individual,  and  educating  users  to  make  more 
efficient  use  of  the  library.  Further,  he  stated  that  tools  alone  will  not  promote  effective 
reuse. 

It  was  made  very  clear  by  interviewees  that  those  who  use  these  libraries  should 
understand  and  manage  the  reuse  process  on  the  most  basic  level,  or  they  risk  the  same 
dangers  as  uncontrolled  cloning.  Moreover,  Drouin  stated  that  good  development 
practices  should  ensure  that  reuse  starts  at  the  design  level,  where  it  is  most  important. 
Drouin  added: 

"You  need  to  be  able  to  demonstrate  at  the  end  of  your  design  how  reuse  will  happen  [in 
the  implementation  of  the  design]. " 
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TRILLIUM  and  Its  Reuse  Criteria  Enjoy  Strong 

Support 


•  Tri-Corporate  cotnmitinent  to  quality  is  a  key  success  factor 

•  Bell's  influence  also  generates  support 

•  Assessment  criteria  need  improvement 

•  Supplier  support  is  demonstrated  by  customer  /  in-house  training 
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NOTES: 


TRILLIUM  AND  Its  reuse  Criteria  Enjoy  Strong  Support  in  Tri-Corporatf 
Oroanizations 


TRILLIUM  implementation  has  been  a  key  objective  for  Bell,  NT  and  BNR  since  the 
program’s  inception  in  1991.  Tri-lateral  support  stems  mainly  from  the  "quality  culture" 
fostered  by  TQM  programs  in  these  organizations.  Other  factors  related  to  this  "quality 
culture"  also  play  a  role  in  generating  support  for  TRILLIUM.  These  include  Bell's 
partnership  approach  to  supplier  relationships,  and  buy-in  to  the  program  by  key 
members  of  the  tri-corporate  organizations.  In  addition  to  a  corporate  emphasis  on 
quality.  Bell's  influence  with  suppliers  plays  a  role  in  their  endorsement  of  the  model. 
Because  of  these  factors,  TRILLIUM  is  now  an  integral  part  of  software  development  at 
NT  and  BNR.  It  has  also  been  implemented  by  the  software  development  organization 
at  Ecole  Polytechnique,  a  research  institution  supported  by  Bell  funding  and  a  supplier 
of  CASE  tools  to  Bell. 
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TRILLIUM  and  Its  Reuse  Criteria  Enjoy  Strong 
Support  (1  of  4) 


Tri-Corporate  Commitment  to  Quality  Is  a  Key  Success  Factor 

•  Commitment  to  continuous  quality  improvement  sets  context  and  provides 
motivation  for  TRILLIUM  and  reuse 

*  Customer  and  supplier  partnership  is  paramount 
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NOTES: 

Tri-Corporate  Commitment  to  Quality  Is  a  Key  Success  Factor  for  TRILLIUM  anH 
Reuse 

Tri-corporate  managers  agreed  that  the  key  success  factor  for  TRILLIUM  and  for 
reuse  is  the  above  mentioned  commitment  to  a  continuous  quality  improvement 
process.  TQM  programs  are  widespread  and  strongly  supported;  all  management 
personnel  have  taken  a  training  program  on  quality,  and  everyone  in  Bell  and  NT  has 
taken  an  internal  quality  training  course.  In  their  view,  the  tri-corporate  enthusiasm  for 
TRILLIUM  is  directly  related  to  the  mindset  engendered  by  these  programs. 

According  to  Coallier,  supplier  maturity  is  necessary  to  secure  the  potential  benefits 
of  the  exercise.  Coallier  explained  that  when  a  good  self-assessment  is  conducted,  gaps 
and  "islands  of  excellence"  in  the  software  development  process  are  identified.  A 
skillful  manager  will  focus  on  these  "islands"  quickly  and  use  them  to  spread  excellence 
throughout  the  organization.  Once  the  gaps  are  bridged,  the  software  development 
process  becomes  much  more  efficient. 

However,  an  immature  self-assessment  imparts  few  benefits.  Coallier  told  us  of  a 
Bell  supplier  self-assessment  outside  of  the  tri-corporate  family  that  was  problematic. 
The  organization  lacked  the  maturity  to  share  constructive  criticisms  of  their  capabilities 
and  instead  sought  primarily  to  achieve  the  highest  possible  score  on  the  TRILLIUM 
scale.  Coallier  stated  that  in  doing  this,  they  overlooked  significant  gaps  in  their 
development  process,  thereby  ignoring  the  ultimate  goal  of  the  exercise.  In  this  case, 
the  self-assessment  was  scrapped  and  TRILLIUM  had  to  be  used  in  audit  mode. 

Tri-corporate  managers  also  stressed  that  creating  a  partnership  between  suppliers 
and  customers  is  of  paramount  importance  for  implementing  TRILLIUM.  An 
antagonistic  relationship  can  only  be  counterproductive  and  will  eventually  negate  the 
effectiveness  of  the  model.  In  addition,  they  stated  that  there  must  be  a  commitment  to 
TRILLIUM  by  key  people  in  the  supplier  organization  who  will  champion  the  program 
and  spread  it  throughout  the  organization. 
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TRILLIUM  and  Its  Reuse  Criteria  Enjoy  Strong 
Support  (2  of  4) 


Bell's  Influence  Also  Generates  Support 

•  Bell  is  Northern  Telecom's  largest  customer  —  $1.3  B  /  year 

•  Other  NT  customers  look  to  Bell  as  a  testbed  for  new  products 

•  Bell  penalizes  suppliers  for  poor  performance 

•  Software/hardware  quality  and  TRILLIUM  programs  under  one  manager 
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Notes: 

Bell’s  Influence  Also  Generates  Support  for  TRILLIUM 

Bell  suppliers  are  motivated  not  only  by  the  potential  competitive  gains  in  the  global 
marketplace  from  using  TRILLIUM  and  the  cultural  factors  as  outlined  above,  but  also 
by  the  leverage  that  Bell  wields  as  their  largest  and  most  influential  customer.  First,  NT 
accounts  for  about  $1.3  billion  of  Bell’s  $3  billion  per  year  in  total  purchases  of  products 
and  services.  Second,  Bell  often  acts  as  a  test-bed  for  new  NT  products.  As  such,  we 
believe  that  other  operating  companies  look  for  Bell’s  acceptance  before  committing  to 
major  purchases.  Third,  Bell  penalizes  suppliers  for  poor  performance.  In  the  past.  Bell 
has  actively  pursued  suppliers,  forcing  them  to  correct  any  defects  or  provide  a  retrofit 
at  no  additional  cost.  While  it  would  be  unfair  to  characterize  these  factors  as  the 
impetus  behind  NT’s  and  BNR’s  acceptance  of  TRILLIUM,  it  would  also  be  unrealistic 
to  assume  ♦^hat  they  carry  no  weight. 
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TRILLIUM  and  Its  Reuse  Criteria  Enjoy  Strong 
Support  (3  of  4) 


Assessment  Criteria  Need  Improvement 

•  Evaluative  questionnaire  is  ambiguous  in  parts,  and  unduly  burdensome 

•  Problems  to  be  resolved  in  Fall  1993  release 

SUdc22 

NOTES: 

Assessment  Criteria  Need  Improvement 

While  TRILLIUM  implementation  has  been  successful,  there  are  problems  with  the 
assessment  criteria.  Managers  at  NT,  BNR,  and  other  organizations  with  TRILLIUM 
experience  have  found  that  some  practices  required  by  the  evaluative  questionnaire  are 
ambiguous,  and  others  are  unduly  burdensome.  To  address  this  problem,  tri-corp)orate 
managers  have  been  trying  to  simplify  the  evaluative  questionnaire  for  future  use.  With 
the  next  release  of  TRILLIUM,  due  in  October  of  1993,  it  is  hoped  that  these  problems 
will  be  eradicated. 
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TRILLIUM  and  Its  Reuse  Criteria  Enjoy  Strong 
Support  (4  of  4) 


Supplier  Support  is  Demonstrated  by  Customer  and  In-house  Training 

•  All  NT  staff  attend  a  course  on  quality 

•  NT  trained  90  staff  as  in-house  assessors 

•  For  Bell,  qualifications  include  extensive  field  experience  and  knowledge  of 
SEI/CMM,  TRILUUM  and  quality 
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Notes: 

Supplier  Support  for  TRILLIUM  Is  Demonstrated  by  Customer  and  In-House 
Training 

NT  training  programs  are  indicative  of  supplier  support  for  TRILLIUM.  They  train 
customers  (other  than  Bell)  who  do  not  have  the  necessary  expertise  to  participate  in  the 
assessment  process.  NT  also  has  internal  programs,  through  which  they  have  trained 
approximately  ninety  people  to  date  as  in-house  assessors. 

NT  makes  this  effort  because  competent  TRILLIUM  assessors  greatly  contribute  to 
the  gains  from  the  model.  In  an  assessment,  the  right  team  of  experts  will  interpret 
practices  in  the  model  to  the  best  advantage  of  the  supplier  organization. 
Inexperienced  people,  however,  will  simply  check  yes  or  no  when  answering  the 
questionnaire,  which  detracts  from  the  value  of  the  exercise. 

According  to  an  experienced  Bell  assessor,  representing  the  customer  in  supplier 
audits  and  self-assessments  requires  extensive  experience  in  the  technical  domain,  a 
knowledge  of  quality,  a  working  knowledge  of  the  TRILLIUM  and  SEI  models,  and 
excellent  management  skills.  Before  starting,  assessors  should  gather  information  on 
the  targeted  organization's  software  products.  They  should  also  keep  a  developer's 
perspective  by  remaining  connected  to  the  software  field  (keeping  their  software 
development  skills  up  to  date). 
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Impact  oi  TRILLIUM  and  Reuse  Criteria  is 
Promising 


•  TRILLIUM  is  key  for  long-term  Tri-corporate  success 

•  Reuse  practices  have  a  positive  impact 

•  Expansion  of  reuse  practices  looks  promising 

•  TRILLIUM  expanding  to  external  suppliers,  Canadian  and  French 
Governments,  ISO  (SPICE) 

-  Externr  i  MIS  suppliers 

-  Weapons  Systems  —  French  MOD 
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NOTES: 

The  Impact  Of  TRILLIUM  and  Its  Reuse  Criteria  Is  Promising 

For  Bell,  there  is  preliminary  data  which  indicate  a  substantial  positive  impact  from 
TRILLIUM  in  terms  of  supplier  performance.  Although  competitive  gains  in  these 
organizations  have  been  attributed  to  many  additional  factors  by  tri-corporate 
managers,  NT  and  BNR  acclaim  the  success  of  TRILLIUM.  For  example,  the  most 
recent  surge  in  growth,  concurrent  with  the  implementation  of  TRILLIUM,  has  been 
attributed  to  the  globalization  of  operations.  At  the  same  time,  TRILLIUM  has  been 
cited  by  these  individuals  as  an  essential  element  in  ensuring  long-term  tri-corporate 
growth. 

Reaction  to  the  reuse  criteria  within  the  model  is  also  positive.  According  to  the 
Manager  of  Transmission  Development  Processes,  TRILLIUM  requirements  have 
attempted  to  formalize  pre-existing  reuse  practices  and  will  help  to  promote  reuse 
within  the  tri-corporate  organizations.  This  statement  is  corroborated  by  other  tri¬ 
corporate  managers.  The  impact  of  TRILLIUM*s  reuse  practices,  however,  has  not  yet 
been  measured  and  is  difficult  to  separate  from  the  pre-existing  reuse  practices. 

TRILLIUM  Is  Described  as  Key  to  Tri-Corporate  Success. 

Tri-corporate  managers  stated  that  TRILLIUM  is  viewed  as  a  means  of  promoting 
future  growth  and  quality.  While  it  is  too  early  to  measure  conclusively,  Coallier  states 
that  participants  in  initial  pilots  appear  to  be  more  user  focused,  responsive  to  customer 
needs,  and  products  are  generally  more  stable.  Management  at  supplier  organizations, 
NT  and  BNR,  stated  that  they  view  the  model  as  a  means  of  achieving  aggressive 
corporate  goals.  They  further  stated  that  TRILLIUM  will  have  a  greater  future  impact, 
when  they  have  implemented  the  practices  at  higher  levels  of  the  model.  This  will  be 
especially  important  as  they  expand  into  the  global  market. 

From  a  different  perspective.  Professor  Pierre  Robillard  at  Ecole  Polytechnique 
stated  that  TRILLIUM  has  been  effective  in  identifying  weaknesses  in,  and  providing 
structure  to,  the  development  process  in  his  organization.  However,  he  did  have 
criticisms  of  the  model.  He  stated  that  TRILLIUM  stresses  process  over  product,  a  point 
that  his  organization  is  currently  studying,  and  he  said  that  some  TRILLIUM  practices 
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are  not  relevant  for  small  development  organizations.  In  his  view,  the  latter  is 
particularly  true  of  the  higher  level  practices  in  the  model,  which  may  not  be  worth 
achieving  on  the  basis  of  cost/ benefit  analysis. 

Reuse  Practices  in  TRILLIUM  Are  Viewed  as  Having  a  Positive  Impact 

Reuse  was  cited  by  a  number  of  interviewees  as  being  an  essential  part  of  any  large 
scale,  modern  software  development  process.  Reuse  was  also  cited  as  a  major  success 
factor  for  tri-corporate  systems.  While  we  were  not  able  to  obtain  documentary 
evidence,  in  recent  years,  there  has  been  a  shift  in  the  way  reuse  is  practiced  in  tri¬ 
corporate  organizations  that  may  result  from  TRILLIUM.  NT  and  BNR  have  placed 
greater  emphasis  on  the  infrastructure  required  for  effective  reuse.  They  are  measuring 
the  long-term  benefits  of  creating  this  infrastructure  and  are  rewarding  developers  for 
producing  reusable  modules,  regardless  of  the  short-term  earning  potential  of  this 
work.  In  fact,  according  to  the  Director  of  Synergy  Architecture,  success  in  developing 
reusable  products  is  now  more  highly  valued  than  producing  code  with  potential  for 
only  short-term  profits.  Further,  greater  emphasis  has  been  placed  on  managing  the 
development  process. 

Expansion  of  Reuse  Practices  in  TRILLIUM  Looks  Promising 

While  several  major  product  lines,  such  as  switching  and  transmission  systems, 
incorporate  advanced  reuse  practices,  only  a  small  set  of  reuse  practices  are  presently 
used  throughout  all  tri-corporate  development  projects.  This  is  a  function  of  the  limits 
of  reuse  knowledge  and  the  difficulty  in  implementing  new  ideas.  In  general,  reuse 
practices  that  are  developed  don't  easily  get  into  "street  practice"  in  the  tri-corporate 
because  the  requisite  infrastructure  for  serious  reuse  oni/  exists  in  pockets.  NT  and 
BNR  have  been  working  to  develop  this  infrastructure  with  the  help  of  the  TRILLIUM 
model.  At  this  stage,  many  of  the  level  4  and  5  TRILLIUM  reuse  practices  are 
projections  which  look  promising  for  future  use  when  technology  exists  to  support 
them.  For  now,  NT  and  BNR  use  only  those  practices  that  are  clearly  defined  and  yield 
a  measurable  benefit. 

TRILLIUM  Is  Expanding  to  External  Suppliers  and  to  Other  Domestic  and 

International  Organizations 

Aside  from  self-assessments  in  the  above  mentioned  organizations,  there  have  been 
a  number  of  supplier  audits  outside  the  tri-corporate  family,  and  there  is  growing 
momentum  to  expand  use  of  the  model  to  assess  external  suppliers  of  management 
information  systems.  Additionally,  TRILLIUM  is  expanding  rapidly  to  other  domestic 
and  international  organizations,  such  as  the  International  Standards  Organization 
(ISO)  and  the  Canadian  and  French  governments. 


15  ISO  is  working  on  a  Software  Process  Improvement  and  Capability  dEtermination  (SPICE)  project,  code 
numbered  JTC1/SC7/WG10.  The  objective  of  the  SPICE  project  is  to  develop  an  international  standard  on 
Software  Process  Assessment 
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Similarities  and  Differences  Provide  Perspective 


•  TRILLIUM  Reuse  Practices  May  Be  Relevant  to  DoD 

•  Differences  Provide  Perspective  for  Future  Strategies 
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NOTES: 

Similarities  and  Differences  Provide  Perspective 

There  are  a  number  of  similarities  and  differences  between  DoD  and  Bell  in  the 
acquisition  and  development  of  software.  The  similarities  stem  from  the  common 
requirements  of  large,  complex  software  systems  that  are  cost  effective  and  assure  high 
quality.  The  differences  exist  primarily  because  of  the  distinctions  in  relationships 
l^tween  Bell  and  its  suppliers  and  DoD  and  its  suppliers.  We  believe  that  the  areas  in 
which  Bell  and  DoD  exhibit  similarities  are  primarily  constants  which  are  necessary  for 
effective  reuse  and  strongly  support  the  relevance  of  this  case  study  for  DoD.  Further, 
dissimilarities,  which  now  exist,  are  variables  which  may  serve  to  clarify  future  DoD 
reuse  strategies. 
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Similarities  and  Differences  Provide  Perspective 

(lof2) 


Technical 

Similarities: 

•  Both  DoD  and  Bell  require  large,  complex,  critical  systems  that  must  quickly 
adapt  to  new  requirements 

•  Both  DoD  and  Bell  benefit  from  COTS  products  and  industry  standards 
Dinerences: 

Bell  /  Suppliers  DoD  /  Suppliers 

BNR  has  built,  controlled  and  used  DoD  has  had  less  success,  e.g.,  ALS 
proprietary  development  and  target  (Ada  Language  System) 
environments 

BNR's  primary  reuse  experiences 
involve  commercial 
telecommimications,  where 
common,  exact  architectiue  and 
interface  standards  are  the  norm 
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DoD  deals  with  an  array  of 
domains,  few  have  this  level  of 
industry-wide  definition  and 
coherence 
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Similarities  and  Differences  Provide  Perspective 

(2  of  2) 


Organizational  and  Regulatory 

Bell  /  Suppliers 

DoD  /  Suppliers 

Tri-corporate  acquisition  policies  reflect 
developing  and  advancing  cooperative 
relationships  with  suppliers,  with  the  end 
goal  of  improving  product  quality 

Cooperative  relationships,  with 
synergistic  participation  among  Govt 
customers  and  contractors,  can  and  do  work, 
but  they  are  not  the  norm 

Relationships  between  DoD  and  suppliers 
are  often  characterized  as  antagonistic 

Formal,  cooperative  relationships,  e.g., 

FFRDCs,  CRADAs  are  not  mainline  supply 
vehicles 

A  single,  unified  Bell  organization  assesses 
and  promotes  quality  for  both  hardware 
and  software 

DoD  is  not  so  centralized 

Bell  purchases  most  telecom  products  from 
one  vendor 

Many  large  DoD  projects  involve  multiple 
vendors  each  with  one  or  more  culhu-es, 
development  environments,  standards,  etc. 

Suppliers  meet  Bell's  requirements  and 
remarket  systems  on  the  international 
market 

Dual  use  is  not  so  prevalent  for  DoD 
suppliers 

National  security  is  not  an  issue 

National  security  is  an  issue 
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Notes: 


Lessons  Learned  from  TRILLIUM  Are  Relevant  for  DoD 

Bell  and  DoD  require  large,  complex  systems  with  similar  reliability  standards  and 
real-time  response  requirements.  Like  Bell's  systems,  many  new  DoD  systems  are 
written  in  a  common,  modular,  high-order  language.  Moreover,  in  the 
telecommunications  industry  and  in  defense  systems,  many  products  have  well-defined 
domains  and  long  life  cycles.  As  noted  earlier  in  the  study,  these  characteristics 
promote  effective  reuse  practices  such  as  those  found  in  TRILLIUM.  Furthermore,  as 
previously  described.  Bell  and  DoD  suppliers  are,  to  a  degree,  similarly  motivated  by 
the  predominant  influence  of  their  largest  customers  (i.e.,  DoD  and  Bell). 

Because  of  the  successes  attributed  to  TRILLIUM  and  its  reuse  standards  by  tri¬ 
corporate  managers,  we  believe  that  the  similarities  in  software  acquisition  and 
development  between  Bell  and  DoD  warrant  further  consideration  of  TRILLIUM  by 
DoD. 
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Differences  Provide  Perspective  for  Future  Strategies 

Despite  the  influence  that  Bell  and  DoD  can  exercise  over  their  suppliers,  there  are 
major  differences  in  organizational  culture  and  approach  to  suppliers  between  these 
organizations.  For  example,  the  relationships  between  tri-corporate  organizations 
(customers  and  suppliers)  are  much  closer  than  those  between  DoD  and  its  contractors, 
which  tend  to  be  more  formal.  Bell  strives  to  build  strong,  mutually  beneficial 
relationships  with  suppliers.  They  emphasize  the  link  between  the  quality  of  suppliers' 
products  and  the  quality  of  their  own  products,  and  use  TRILLIUM  assessments  to 
strengthen  supplier  software  development  performance^  This  will,  in  turn,  have 
positive  repercussions  for  their  own  systems. 

Conversely,  "arms  length,"  and  in  some  cases,  adversarial  relationships,  exist 
between  the  DoD  and  its  suppliers.  This  can  be  an  impediment  to  using  a  program  like 
TRILLIUM.  According  to  Bell  managers,  TRILLIUM  requires  partnering  of  customers 
and  suppliers  to  achieve  the  full  benefits  of  the  model.  TRILLIUM's  greatest  impact  is 
in  helping  suppliers  develop  better  software  development  practices  through 
assessments,  not  in  auditing  these  practices  and  setting  arbitrary  standards. 

Based  on  this  information,  we  believe  that  an  environment  of  cooperation  is 
necessary  to  effectively  implement  practices  or  to  adapt  TRILLIUM  or  a  similar  model 
for  DoD  purposes.  If  cooperative  relationships  cannot  be  developed  between  DoD  and 
its  suppliers  and  the  model  is  instead  imposed  on  suppliers,  it  may  inhibit  the  potential 
opportunities  and  benefits  associated  with  a  successful  application  of  reuse  in  the 
TRILLIUM  model. 

Other  major  difference  between  Bell  and  DoD  exist  in  standards  for  software 
architecture  and  regulations  in  the  defense  and  telecommunications  industries.  The 
telecommunications  industry  is  advantaged  by  the  presence  of  industry-wide  standards 
such  as  Bellcore  TR303  and  TR308,  which  are  consensual  and  unambiguous.  Bellcore 
defines  industry  standards  for  software  architecture,  and  suppliers  work  to  meet  these 
standards.  In  contrast,  the  defense  contracting  industry  deals  with  a  vast  array  of 
systems,  most  of  which  do  not  have  well-defined,  shared  architectures.  Further, 
telecommunications  firms  are  not  bound  by  stringent  procurement  and  other 
government  regulations  that  exist  in  the  defense  contracting  industry. 

Moreover,  Bell  and  DoD  supplier  market  situations  differ.  NT  and  BNR  do  not 
deliver  custom-built  software  to  Bell  for  its  large  switch  software.  They  deliver  the 
same  products  to  other  large  commercial  clients,  and  realize  that  TRILLIUM  has  the 
potential  to  facilitate  competitive  gains  in  all  markets.  Many  DoD  suppliers,  however, 
develop  products  exclusively  for  DoD  according  to  specifications.  In  this  situation, 
there  is  less  incentive  for  suppliers  to  implement  a  reuse  practice  like  TRILLIUM,  as  it 
will  not  help  their  competitive  situation  in  other  markets.  We  believe,  however,  that 
with  the  impending  conversion  to  commercial  applications  of  DoD  supplier  technology 
(dual-use  technology), this  discrepancy  may  soon  begin  to  be  ameliorated.  Under  this 
condition,  a  model  like  TRILLIUM,  containing  reuse  criteria,  could  assist  in  stimulating 
the  conversion  process. 


DoD  and  a  number  of  other  Federal  agencies  have  developed  a  government-wide  program  to  focus  on 
technologies  of  critical  importance  to  both  national  security  and  the  national  economy. 
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Key  Factors  Contributing  to  Reuse  Success 

•  Assimilating  reuse  criteria  into  a  capability  assessment  model  and  total  quality 
management  process 

•  Providing  an  atmosphere  of  participatory  management  within  a  defined  and 
controlled  group  of  suppliers  and  users 

•  Selecting  projects  that  have  characteristics  (large,  complex  systems  with  stable, 
welNdeflned  domains  and  long  life  cycles)  conducive  to  securing  the  benefits  of 
reuse 

•  Using  proper  development,  analysis,  architecture,  tools  and  techniques  in  a 
reuse-oriented  management  structure 

•  Adapting  reuse  approaches  to  meet  evolving  needs 


Slide  28 

Notes: 

Conclusions 

Bell  developed  and  implemented  TRILLIUM  and  its  reuse  practices  within  a  close 
supplier  network.  This  gave  them  a  level  of  cooperation  and  trust  that  may  be  difficult 
to  develop  widely  for  DoD  and  its  contractors.  Nevertheless,  reuse  practices  in 
conjunction  with  a  defined  quality  control  process  is  considered  by  tri-corporate 
managers  to  be  a  major  factor  in  developing  better  quality  software  systems  at  less  cost. 

In  our  view,  the  key  reuse  factors  contributing  to  this  success  are: 

•  assimilating  reuse  criteria  into  a  capability  assessment  model  and  total  quality 
management  process 

•  providing  an  atmosphere  of  participatory  management  within  a  defined  and 
controlled  group  of  suppliers  and  users 

•  selecting  projects  that  have  characteristics  (large,  complex  systems  with  stable, 
well-defined  domains  and  long  life  cycles)  conducive  to  securing  the  benefits 
of  reuse 

•  using  proper  development,  analysis,  architecture,  tools  and  techniques  in  a 
reuse-oriented  management  structure 

•  adapting  reuse  approaches  to  meet  evolving  needs 
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6.5.2.7  A  formal  process  is  used  to  create  and  release  One-ups  [S0  SCM  Activity  8], 

6.5.2.8  A  Software  Confiwration  Management  (SCM)  organization^function  exists  for  each 
project,  responsible  for  creating  and  managing  the  software  ISsraiy  for  the  proiect  [SEl 
SCM  Ability  21  (IEEE  Std.  828  2.2)  [Trillium]. 

6.5.2.9  Source  code  is  urxier  SCM  control  [Trillium]  [Bellcore  TR-TSY-000179  2.5  8  & 

2.10.1.2]. 


Levels 

6.5.3.1  User  requirements,  specification  documents,  design  documents,  source  code  and  test 
cases  are  under  SCM  control  [SEl  SPE  Activity  11]  [Bellcore  TR-TSY-000179 
2.10.1.1]. 

6.5.3.2  There  is  full  forward  and  backward  traceability  between  all  configuration  items  (e.g  de¬ 
sign  documents  forward  to  code  units,  design  documents  beocward  to  specifi^on 
documents)  [SEl  SPE  Activity  11]  [Bellcore  TR-TSY-000179  2.4.1]  (Ref.  IEEE  Std. 


Level  4 

6.S.4.1  All  development  tools  are  under  SCM  control  [Trillium]. 


UvelS 

6.5.5.1  Development  history  (e.g.  design  decisions,  design  rationale)  is  captured  and  mairv- 
tained  under  SCM  control  [TriHium]. 


Level  2 


6.6.2.1  -  Cloning  (i.e.  copying  and  modifying)  of  existing  software  units  in  new  designs  is  tracked 

[Trillium]. 

6.6.2.2  Tools  are  provided  to  aid  propagation  of  changes  from  one  software  unit  to  cloned  or 
cloned  from  units  [rrillium]. 

6.6.2.3  A  formal  procedure  for  the  selection,  verification  &  vaOdation  and  tracking  of  third-party 
software  components  is  in  place  [Trillium]  [ISO  9000-3  6.8]. 


Level  3 

6.6.3.1  Pre-developed  and  certified  source  code  templates  are  provided  in  a  template  library 
as  a  basis  for  cloning  [Trillium]. 

6.6.3.2  Pre-developed  and  certified  source  code  units  £U’e  provided  in  a  software  component 
library  [Triillum]. 
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6.6.3.3  A  component  development  function  exists  to  develop,  certify  and  mairttain  the  items  in 
the  template  and  component  Ibrary  fTnilium]. 

6.6.3.4  SpeciTic  steps  are  included  in  the  organization’s  standard  software  development  proc¬ 
ess  for  the  purpose  of  maximizing  comporrent  re-use  [Trillium]. 

6.6.3.5  Re-use  of  pre-deveioped  components  is  measured,  encouraged,  and  rewarded  by  the 
organization’s  reward  system  (T riilium]. 


Level  4 


6.6.4. 1  Pre-developed  and  certified  designs  are  provided  in  a  software  component  library  for 
the  use  of  developers  [Trillium j. 

6.6.4.2  CASE  tools  are  provided  to  support  re-use  of  software  components  in  new  product  de¬ 
velopments  [Trillium]. 


Levels 

6.6.5.1  Knowledge  based  CASE  tools  are  provided  to  automate  re-use  of  components  in  new 
product  developments  [Trillium]. 


6.7  Rellabnity  Management 
Level  2 

6.7.2.1  System  and  senrice  wide  reliability  arxi  maintainability  plans  and  objectives  are  estab¬ 
lished  [Trillium]  [I  EC  300]. 

6.7.2.2  System  Availabinty  (unavailability)  is  defined  at  the  product  definition  phase,  including 
Failure  Definition/Ciassification  (Critical,  m^r,  mirxjr)  and  Operational  Modes  Defini¬ 
tion  (nort-stop,  missiott-oriented,  batch,...)  [Triilium]  [lEC  300]. 

6.7.2.3  Software  [Un]  availabiRty  is  apportioned  at  the  product  definition  phase  including  S/W 
[Un]availabiiity  due  to  Enhancements  (upgrades),  S/W  [Un]avaiiability  due  to  ^lures 
and  S/W  Failure  Intensity  requirement  definition  prer  failure  classes  [TrilGum]. 

6.7.2.4  The  following  data  is  collected  arxi  processed  (Ref.  IEEE  Std.  982.1  &  982.2); 

a)  Test  Log  (Test  Execution  Tune), 

b)  Failure  Log, 

c)  Installation  Date/Time  Recording, 

d)  Cumulative  Execution  Time  Logging  per  release, 

e)  Failure  recording  and  classification, 

f)  Failure-related  down  time  recording, 

g)  Enhancement-related  down  time  recording. 
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